GROUP-WISE VIDEO CONFERENCING USES 
3D-GRAPHICS MODEL OF BROADCAST EVENT 


1 FIELD OF THE INVENTION 

2 The invention relates to a method and a system for enhancing broadcasting with a service that 

3 enables interaction among multiple end users that are geographically distributed. 

4 

5 BACKGROUND ART 

6 Examples of communication involving multiple users are a broadcast and a conference. The 

7 broadcast is typically a one-to-many exchange of pre-recorded or real time information without 

v n 8 interaction of the receiving party with the broadcasting process. A conference is a form wherein 

O 

Ul 9 dialogues are typically real-time and dynamic in the sense that receiver and sender interact and 

i t £ 

^ io frequently change their roles and determine the information exchanged. 

1 1 Examples of multimedia methods and systems in a multiple-user, interactive virtual environment 

J 12 that enables conferencing are discussed in, for example, U.S. patent applications of Philips Electronics, 

6 

Q 13 serial no.'s 08/373,737 (PHN 14,719); 08/597,439 (PHN 15,187) and 08/ 828,468 (PHN 15,769), 

in 

p 14 herewith incorporated by reference. In an implementation of the known systems, a cable-TV network 

15 connects multiple users to an application server. The server provides the graphics for a virtual 

16 environment via teletext pages, each respective one supplying the graphics data for a respective user. 

17 The telephone network is being used for communicating to the server commands from the user 

18 controlling his/her telephone keys to control his/her graphics avatar in the virtual environment. The 

1 9 teletext graphics pages get updated under control of the commands entered by the user. The telephone 

20 network is also being used to enable communication between users under control of a chat box 

2 1 application run on the server. 
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2 OBJECT OF THE INVENTION 

3 Currently, broadcast and conference modes of communication systems are implemented in an 

4 independent manner with separate applications, e.g., a television broadcast of a sports program and a 

5 telephone or video conference between sports experts being consulted via an audio or video link during 

6 a live broadcast of the sports event, while the conference is being broadcasted. 

7 It is an object of the invention to provide a new interactive environment, and to broaden the 

8 scope of TV broadcast services. It is further object to integrate broadcast and conferencing. 
9 

q 10 SUMMARY OF THE INVENTION 

0 1 1 To this end, the invention provides a method of controlling communication to multiple end users, 

5g 12 different users residing at geographically different locations. The method comprises, in a broadcasting 

± 

|; 13 mode, broadcasting content information for receipt by the end users, and, in a conferencing mode, 

14 enabling interconnecting at least one subset of the end users through a network and enabling interaction 

ess 

p 1 5 between the end users of the subset. The method enables switching between the broadcasting mode and 
C 

y3 16 the conference mode. 

W 

17 The method of the invention thus integrates broadcasting, e.g., TV broadcasting, with 

18 conferencing, and controls the switching between these modes. The invention enables users to discuss 

1 9 certain events that occur in the broadcasting. Preferably, certain events in the broadcast mode trigger the 
2 0 switching to the conference mode. Preferably, the conference mode is enhanced with 3D-graphics 
2 1 models of the triggering events in order to serve as a basis for discussion in groups that are smaller than 
2 2 the population of the audience attending the broadcast. Software for real-time conversion of video into 
23 3D graphics is commercially available. 
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1 BRIEF DESCRIPTION OF THE DRAWINGS 

2 The invention is explained by way of example and with reference to the accompanying drawings, 

3 wherein; 

4 Fig. 1 is a diagram of a known broadcasting system; 

5 Fig.2 is a diagram of a system in the invention; and 

6 Figs 3-5 are diagrams illustrating the method of the invention 

7 Throughout the figures, same reference numerals indicate similar or corresponding features. 


8 

9 PREFERRED EMBODIMENTS 

p 10 Known broadcast system 

p 1 1 Fig. 1 is a block diagram with the main components of a conventional broadcast system 100 for 

W 12 downloading information to the end users. System 100 has a camera 101, a server 102 and multiple 

4™ 

J 13 clients, of which only client 104 is shown in order to not obscure the drawing. Server 102 is typically 

p 14 part of professional studio equipment. Client 104 makes accessible to the end user the information 

q 15 broadcasted by server 102. Typically, client 104 comprises consumer electronics equipment. 

C 

g 16 Server 102 comprises a real-time encoder 108, a storage 1 10, a mixer 1 12, a transport encoder 

17 1 14, and a transmitter 1 16. Mixer 1 12 mixes the data supplied by encoder 108 and storage 110. Storage 

18 110 stores pre-recorded video or graphics data. Real-time encoder 108 encodes the video captured by 

19 camera 101 into a format suitable for the mixing with the data supplied by storage 110. Encoder 1 14 
2 0 encodes the stream into the MPEG-2 TS format. Preferably, the mixing is carried out under control of 
2 1 studio personnel, e.g., the local editor. 
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1 Client 104 comprises a set-top box 118 and a television apparatus 120. Set-top box 118 

2 comprises a receiver 122 and a decoder 124. Transmitter 1 16 in server 102 communicates with receiver 

3 1 12 of client 104 using an MPEG-2 Transport Stream (TS) format. 
4 

5 Broadcast and conferencing system , 

6 Fig. 2 is a block diagram with the main components of a system 200 of the invention. System 

7 200 integrates broadcasting with conferencing. The system architecture is discussed with reference to 

8 Fig.2, its operation is explained further with reference to Figs.3-5. 

9 System 200 presents an integrated approach to broadcast and conferencing modes under software 
f* s 1 0 application control. This approach allows switching the users between the broadcast and conference 

; 

Tp=r 

O 1 1 modes. The switching can be controlled by the server, by the end user, or by both. The conference is 

W 12 triggered by the context set by the broadcast mode. In the conference mode, the clients receiving the 

j 13 broadcast are split into smaller groups for multi-user communication, e.g., discussions about a 

p 14 controversial action during a sports event broadcasted. At the end of the conferencing, the users in a 

p 15 group join the broadcast program in a suitable manner. In order to set the context for the conferencing, 

yl 1 6 audio, video and 3D graphics models are generated based on the content of the broadcast programs, and 

i 

17 are transported to the users. For the group- wise conferencing, the users' clients employ speech, audio, 

18 video, and graphics data, and use streaming protocols, and distributed shared object support. 

19 A service provider can introduce above functionality in an evolutionary manner. This evolution 
2 0 can proceed in a variety of ways. For example, one could introduce this functionality of switching 

2 1 between broadcast and conferencing modes to all users in a stepwise manner, or to a small set of users, 

22 e.g., for professional application. The small set of users is then a set of expprts who need to establish a 
2 3 multi-user collaboration/communication, e.g., a set of soccer experts who are located at geographically 
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1 different sites, and who are called in during a broadcast to give their expert opinion on a particular event 

2 that occurred to the broadcast soccer match. This collaboration/communication is then broadcasted to 

3 all other users. Note that this approach goes beyond current practice of consulting a remotely located 

4 expert with an audio or video link. 

5 System 200 comprises a server 202 and multiple clients, of which only a single one, client 204, 

6 is shown in order to not obscure the drawing. The clients reside at different locations. In addition to 

7 components 101, 108-116, 122 and 124, mentioned above, server 202 comprises other components. 

8 Similarly, in addition to receiver 122 and decoder 124, client 204 comprises other components. The 

9 additional components manage the broadcast mode and conference mode as explained below. 

1 0 Server 202 comprises a model generator 206, an event-triggered controller 208, a unit 210 that 

11 manages the Session Description Protocols (SDP) and the Session Announcement Protocols (SAP). 

12 These protocols are known Internet protocols that support multicasting. For more information, see for 

1 3 example, the paper "How IP Multicast Works, An IP Multicast Initiative White Paper" of authors Vicki 

14 Johnson and Marjory Johnson, Stardust Technologies, Inc., as available on the web at: 

1 5 http://www.ipmulticastxom/comm and its literature references. 

16 SDP describes multimedia sessions for the purpose of session initiation, such as invitations and 

17 announcements. SAP also is meant to ensure authentication and privacy. 

1 8 Server 202 describes the groups thus formed using the description protocol SDP and informs the 

19 clients of the groups being formed by using the announcement protocol SAP. The clients respond by 
2 0 joining a particular group or by waiving to do so. Joining a group automatically activates the conference 
2 1 software application required for enabling the user to participate in the group activities as discussed 
2 2 below. System 200 further comprises a data base 212 with identifications of the clients, such as 
23 of client 204, and information regarding the preferences, authorization, etc. of the clients, in order to 
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2 users 
3 
4 
5 


1 form the groups for the conferencing mode. This information is based, e.g., upon a query among the 
carried out in advance. Model generator 206 is coupled to camera 101 via a server input 203, 
storage 1 10 and event-triggered controller 208. Gene,*, 206 generates 3D graphics models, e.g., in a 
VRML format, of the video da* suppliedby camera 101, or modifies the 3D graphics models stored in 
storage 110. Software for real time conversion of video into 3D graphics is known, for example, as a 

7 triggers generator 206 to create a 3D graphics model in response to the occurrence of a certain event. 

8 The event corresponds to a pre-programmed condttion or is a manual input by, e.g, a sports 
, commentator or studio persotmel, during the broadcasting. Controller 208 also tnggers the formation 

10 of groups of clients, which could be for entering a conference mode, or for watching a conference 

12 Client 204 has a set-top box 214 that comprises a software application 216 for control of a 

1 3 conferencing mode of mis particular client 204. Conferencing modes are further explained below and 

15 communication between client 204 and the other clients in the group to which it is assigned. To this end, 

16 application 216 communicates with data base 212. Client 204 receives via a serve, output 207 and a 

IS an interne. Protocol (IP), or via the broadcast channe! with IP over MPEG-2 TS. Application 216-, 

1, i ^ M ~**^^l^ i *^*"™* ,M " t " W " 

2 „ is only permitted ,„ watch the 3D scene torn different points of view, or also to modify the scene, e.g, 

21 - 

22 Within this context, generator 206 ,s preferably capable of generating different models for different 

23 group, Application 216 controls a 3D renderer 218 that comprises, for example, a VRML browser. 
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1 Decoder 1 24 and Tenderer 2 1 8 are connected to a compositor 226 that processes the input to prepare for 

2 display and play-out to the user at display 120. Compositor 226 is also connected to an output of 

3 A/V/Speech coders 228. A/V streaming protocols 226 enable efficient audio/video data transport 

4 between the clients via realtime communication channels 225, here through the Internet, in the 

5 conferencing mode. A/V/Speech coders 228 take care of the encoding of the A/V/Speech input of client 

6 204 via a microphone 230 and of the decoding of the stream received from the other clients. 

7 Client 204 and the other clients in the same group as client 204 interact via the Internet/Multicast 

8 Routers 220. This interaction is supported locally, at client 204, by a world model and distributed shared 

9 object support protocols (S.O.S.) 222, in order to maintain overall consistence in the 3D model when 
□ 1 0 being manipulated by authorized users. To this end, a user input device 232, e.g., a joy-stick, is provided 
O 1 1 at authorized client 204 for modifying or manipulating in another manner the 3D model via application 

m 

W 1 2 216. A Group Management unit 234 handles group management, authentication access control and 

5 13 subscription issues such as payment. Unit 234 is, for example, part of application 216 or is a separate 

p 14 application, or is implemented with a smart card reader. Unit 234 receives the relevant control 

H 15 information from server 202 via an input 233. 

gO 16 Note that components 124, 216, 218, 222, 224, 226, 228 and 234 may all be implemented in 
03 

17 software, 

18 Operation is as follows. The transition from a broadcast mode to a conference communication 

19 mode is triggered by an event. This triggering can be automatic or manual, determined by a sports 
2 0 commentator for a live broadcast or by studio personnel for a pre-recorded program. On the event of a 
2 1 trigger from controller 208, model generator 206 creates the 3D graphics models, possibly different ones 
2 2 for different groups of users. 
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1 Fig. 3 illustrates the transitions between a large group 302 watching the broadcast and smaller 

2 groups 304, 306, 308 formed out of larger group. 

3 Fig.4 illustrates a more detailed scenario, wherein a large group 402 comprises a group 404, a 

4 group 406 and a group 408. The users in group 404 switch between the broadcast mode reception and 

5 the conference mode and remain passive in the sense that they merely receive information and do not 

6 interact actively. The users in group 408 are divided among a plurality of smaller groups 410, 412, 

7 414, each not necessarily of the same users during the session, for participating in the conferencing. The 

8 users in group 406 form a panel whose conference is merged with the broadcasting to all users who want 

9 to receive this. 

q 10 Fig.5 illustrates a refinement on the scenario of Fig.4. It is possible that not all users can or want 

□ 1 1 to enter the conference mode, either for attending a conference in a small group or for viewing the 

V s 

W 12 conference of another group, e.g., of the soccer experts group. For example, not all users capable of 

g 13 receiving the broadcasted information have the equipment supporting the switching between the 

p 14 broadcast mode and the conferencing mode. Under this scenario, a group 502 stays out of and is not 

15 hampered by the switching scenario. 

$ 


S : \VK\ PU3 1VKA0 . VKR 


8 


